Embodiments disclosed herein relate to FPGA-based coherent accelerators, and more specifically, to emulating memory mapped input/output (I/O) for FPGA-based coherent accelerators in an error state.
Conventional hardware accelerators (e.g., PCI-based accelerators) perform operations requiring direct memory access (DMA) via a stack that includes a number of layers, providing user applications with access to the hardware accelerator. The hardware accelerator directs data to a physical memory address from a storage device attached to the hardware accelerator. The operating system sets up DMA to corresponding pages of physical memory for the application (i.e., a buffer for the application). Doing so allows the hardware accelerator to arbitrate on a connected bus (e.g., a PCI bus) to transfer I/O operations and system calls to the storage device via a series of SCSI commands executed in the hardware accelerator.
Some FPGA-based hardware accelerators provide an application with direct access to the hardware accelerator. For example, an FPGA-based coherent accelerator allows an application to execute SCSI commands directly within application memory. In contrast to traditional hardware accelerators that need physical addresses to execute bus commands, coherent accelerators use effective addresses to issue bus commands to an attached storage device. As a result, an operating system does not need to perform actions that are typical (and computationally expensive) to a traditional hardware accelerator, such as translating effective addresses to physical addresses, which requires steps such as pinning memory pages to prevent page faults. A coherent accelerator translates effective addresses to real addresses while accelerating a function. Therefore, the operating system, via the coherent accelerator, allows page faults to occur, handling the page faults such that the accelerator may continue to access application memory. A coherent accelerator translates effective addresses to real addresses while accelerating a function. Therefore, the operating system, via the coherent accelerator, allows page faults to occur, handling the page faults such that the accelerator may continue to access application memory. This approach greatly reduces the length of instructions required to set up a DMA path for data transfer.
User applications can also control FPGA-based coherent accelerators via memory mapped I/O (MMIO). MMIO generally uses the same address bus to address both memory and I/O devices (e.g., FPGA-based coherent accelerators, etc.). The I/O device's registers are mapped to address values in the memory space used by the applications. As such, the instructions that are used to access the physical memory can also be used to access the I/O devices. As an example, when a user application accesses an address, the address may refer to a portion of physical RAM or may refer to registers on the I/O device.
In some cases, FPGA-based coherent accelerators can implement an error recovery protocol to prevent applications from crashing or performing invalid actions when the accelerators encounter certain events, such as, triggering a reset of the accelerator, encountering errors on the link, and other types of errors that require recovery. Once error recovery is initiated, applications may continue attempts to access the accelerators via MMIO without knowing that the accelerators have encountered an error. Such behavior is possible because applications can generally directly access FPGA-based coherent accelerators without involving device drivers or the kernel (unless there is a page fault or error interrupt). Accessing coherent accelerators in this manner, however, can cause undesirable behavior, such as, e.g., applications crashing, operation system crashes, and the like.